Communication in Peer Reviews: Effects of Computer Interfaces, Problem Equivicality, and Initial Annotations

Author: Patricia Gail Wojahn
Degree: Ph.D. in Rhetoric, Carnegie Mellon University, 1999

Throughout the collaborative process, practices such as passing rough drafts and individually annotating them are widespread, even among those working in the same office building, department, or school (Fleming, Werner, Sinsheimer-Weeks, & Kaufer, in press; Kraut, Galegher, & Egido, 1987-1988). Professional colleagues have been found to rely increasingly on computer technologies for communicating and collaborating at a distance, particularly after they have set preliminary goals and made initial plans for shared projects (Carley & Wendt, 1991; Kraut, Egido, & Galegher, 1990). Yet little is known about the effects of the interfaces that are being used to support such work. With the continued influx of new communication technologies, the medium as a component of the collaborative process requires more attention than ever before.

My dissertation study was designed to examine the role of particular features of technologies in particular types of collaborative, communicative acts. To explore peer reviews in particular, I focus on communications concerning specific types of problems in the text and specific types of problems in comments from previous annotators. I examine the extent to which peer reviewers address (1) the more equivocal, often substantive, problems that, if resolved, can have significant impact on a text, and (2) the less equivocal, typically more straightforward problems that, if resolved, tend to affect the text at a more local level. I also examine the extent to which peer reviewers address situations reflecting three typical problem cases we face when working with annotations from a reviewer: valid annotations, invalid annotations, or no annotations where we perceive a problem exists.

The study measures effects of specific interfaces on these different types of sub-tasks. Among the various interfaces found in computer technologies for communicating, three types of screen displays were selected for participants to use in the study. The first is the "Interlinear" Interface, a standard display as found in Microsoft Word, currently one of the most widely used word processors, a display in which reviewers embed comments into a draft itself and differentiate comments from the primary text through font features such as bold-face type or brackets. The second, the "Aligned" Interface, displays text and comments in columns used as online margins. The Aligned screen desplay was selected to test a column-type display with a design informed by research on collaborative writing. The final screen display, the "Split-Screen" Interface, incorporates Microsoft Word's annotation feature, one in which the screen is separated into two frames, with markers in the primary text corresponding to annotations presented in the bottom frame, in a display similar to the formats common to footnote presentation.

The most interesting findings from this study include (1) the tendency for people using the Split-Screen interface to make fewer comments than did their counterparts using interfaces with comments positioned more closely to the relevant text; (2) the tendency for participants to comment on problems already discussed by an initial reviewer; (3) the trend for participants to err most when addressing an initial reviewer's Invalid Annotations about text problems; and (4) the tendency for participants addressing High Equivocal problems to use fewer words when their annotations were incorrect than when their annotations were considered accurate.